Information processing apparatus and non-transitory computer readble medium storing program

ABSTRACT

An information processing apparatus includes a processor configured to set a first reference object, and if a second reference object identical or similar to the first reference object is recognized, virtually display a target object in relation to the second reference object. The target object is recognized in advance together with the first reference object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a Continuation of application Ser. No. 17/060,410 filed Oct. 1,2020, which is based on and claims priority under 35 USC § 119 fromJapanese Patent Application No. 2020-039187 filed Mar. 6, 2020. Thedisclosures of the prior applications are hereby incorporated byreference herein in their entirety.

BACKGROUND (i) Technical Field

The present disclosure relates to an information processing apparatusand a non-transitory computer readable medium storing a program.

(ii) Related Art

There is known a technology for displaying an image of an object in avirtual space in relation to an object in a real space.

Japanese Unexamined Patent Application Publication Nos. 2011-28309 and2013-168167 describe apparatuses in which a cursor or an object movingout of a display area of a PC or the like is displayed as a virtualobject.

Japanese Unexamined Patent Application Publication Nos. 2013-93031 and2010-257359 describe apparatuses in which data that is not displayed ina display area of a PC is set as a virtual object and laid over, forexample, a finger of a user's hand.

SUMMARY

Aspects of non-limiting embodiments of the present disclosure relate tovirtual reproduction of an object recognized together with a referenceobject in relation to an object identical or similar to the referenceobject.

Aspects of certain non-limiting embodiments of the present disclosureaddress the above advantages and/or other advantages not describedabove. However, aspects of the non-limiting embodiments are not requiredto address the advantages described above, and aspects of thenon-limiting embodiments of the present disclosure may not addressadvantages described above.

According to an aspect of the present disclosure, there is provided aninformation processing apparatus comprising a processor configured toset a first reference object, and if a second reference object identicalor similar to the first reference object is recognized, virtuallydisplay a target object in relation to the second reference object. Thetarget object is recognized in advance together with the first referenceobject.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present disclosure will be described indetail based on the following figures, wherein:

FIG. 1 is a block diagram illustrating the configuration of aninformation processing system according to a first exemplary embodiment;

FIG. 2 is a flowchart illustrating an image acquisition process;

FIG. 3 illustrates a display screen;

FIG. 4 is a flowchart illustrating a virtual object display process;

FIG. 5 illustrates a display screen;

FIG. 6 is a flowchart illustrating an image acquisition process;

FIG. 7 is a flowchart illustrating a virtual object display process;

FIG. 8 is a flowchart illustrating an image acquisition process;

FIG. 9 illustrates a display screen;

FIG. 10 is a flowchart illustrating a virtual object display process;

FIG. 11 schematically illustrates real objects in a real space;

FIG. 12 illustrates a display screen;

FIG. 13 schematically illustrates real objects in the real space;

FIG. 14 schematically illustrates a real object and a virtual object;

FIG. 15 is a block diagram illustrating the configuration of aninformation processing system according to a second exemplaryembodiment;

FIG. 16 is a flowchart illustrating an image acquisition process;

FIG. 17 is a flowchart illustrating a virtual object display process;

FIG. 18 schematically illustrates real objects in the real space; and

FIG. 19 illustrates a display screen.

DETAILED DESCRIPTION First Exemplary Embodiment

An information processing system according to a first exemplaryembodiment is described with reference to FIG. 1. FIG. 1 illustrates anexample of the hardware configuration of the information processingsystem according to the first exemplary embodiment.

The information processing system according to the first exemplaryembodiment displays a virtual object in a real space by using atechnology such as augmented reality (AR) or mixed reality (MR). Forexample, AR is employed and AR glasses 10 are used as an informationprocessing apparatus in the form of glasses. The AR glasses 10 have aphoto-shooting function and a communication function. The AR glasses 10include various sensors such as a sensor that acquires positionalinformation of the AR glasses 10 (e.g., a Global Positioning System(GPS) sensor), and a gyroscope that detects an azimuth and a posture.

An object in the real space may hereinafter be referred to as “realobject”. The virtual object is a concept provided in contrast with thereal object. For example, the virtual object is an image, a text, agraphical object, or other information. The image is a still image, avideo, or a combination thereof.

The AR glasses 10 include a display 12 in the form of glasses, and acamera 14 that shoots the real space along a viewing direction of a userwearing the AR glasses 10. An image shot by the camera 14 is hereinafterreferred to as “camera image”. The display 12 may be a see-through ornon-see-through display. The display 12 virtually displays an object inrelation to a real object by AR. The virtual display of an object inrelation to a real object means that a virtual object that is an exampleof an image showing the object is displayed on a screen of the display12 while being laid over the real space. For example, a right-eye screenand a left-eye screen are generated and displayed on right and leftdisplays based on parallax of eyes of the user. When the user views thereal space through the display 12, the virtual object is laid over thereal space as if the object actually existed in the real space.

A head-mounted display (HMD) using AR or MR glasses or an HMD using MRmay be employed in place of the AR glasses 10. A mobile terminal havinga camera, such as a smartphone, a mobile phone, or a gaming device, maybe employed in place of the AR glasses 10. In this case, the camera mayshoot the real space and a virtual object may be displayed on a screenof the mobile terminal while being laid over an image of the real space.

The information processing system according to the first exemplaryembodiment includes an information processing apparatus 16. Examples ofthe information processing apparatus 16 include a desktop or notebookpersonal computer (hereinafter referred to as “PC”), a tablet terminal,a smartphone, a workstation, and a server. The information processingapparatus 16 has a communication function to exchange information withthe AR glasses 10. The information processing apparatus 16 may beinstalled in the AR glasses 10.

For example, the information processing apparatus 16 includes acommunication device 18, a user interface (UI) 20, a memory 22, and aprocessor 24.

The communication device 18 is a communication interface (e.g., anetwork interface) having a communication chip, and has functions oftransmitting data to and receiving data from other apparatuses.

The UI 20 includes at least one of a display or an operator. Examples ofthe display include a liquid crystal display and an EL display. Examplesof the operator include a keyboard, input keys, and an operation panel.The UI 20 may be a touch panel having both a display and an operator.

The memory 22 has one or more data storage areas. Examples of the memory22 include a hard disk drive, various memories (e.g., a RAM, a DRAM, anda ROM), other storages (e.g., an optical disc), and combinationsthereof. For example, the memory 22 stores camera images, object images,and virtual objects.

The processor 24 controls operations of individual parts of theinformation processing apparatus 16. The processor 24 may include amemory.

For example, the processor 24 sets a first reference object and, if asecond reference object identical or similar to the first referenceobject is recognized, causes the display 12 of the AR glasses 10 tovirtually display a target object in relation to the second referenceobject. The target object is recognized in advance together with thefirst reference object. The first reference object, the second referenceobject, and the target object are real objects. The virtual display ofthe target object means that a virtual object showing the target objectis displayed on the screen of the display 12 while being laid over thereal space.

For example, the processor 24 may recognize the first reference object,the second reference object, and the target object by processing acamera image generated by the camera 14, or based on detection resultsfrom various sensors instead of the camera 14.

Examples of the second reference object similar to the first referenceobject include an object having a shape similar to the shape of thefirst reference object, and an object having a function similar to thefunction of the first reference object. For example, the object having afunction similar to the function of the first reference objectcorresponds to a tablet terminal having a function equivalent to thefunction of a notebook PC serving as the first reference object. Thesecond reference object similar to the first reference object may be anobject in a category identical to that of the first reference object.For example, the object in a category identical to that of the firstreference object corresponds to a notebook PC having a shape differentfrom the shape of a notebook PC serving as the first reference object.For example, the processor 24 processes a camera image generated by thecamera 14 to recognize a category of an object in the camera image. Theuser may set the category of the object with reference to the cameraimage.

Examples of processes to be performed by the information processingsystem according to this exemplary embodiment are described below indetail.

EXAMPLE 1

Example 1 is described below.

A process for acquiring images of the first reference object and thetarget object is described with reference to FIG. 2. FIG. 2 is aflowchart illustrating the process.

First, the camera 14 of the AR glasses 10 shoots an image and theprocessor 24 acquires an image of the first reference object from thecamera 14 (S01). The processor 24 stores the image of the firstreference object in the memory 22.

The user may select the first reference object, or the processor 24 mayrecognize an object viewed by the user for the longest time as the firstreference object.

An example of selection of the first reference object by the user isdescribed below.

In response to the user giving an instruction to execute a referenceobject selection mode by gesture, voice, or operation for the UI 20, theinformation processing system enters the reference object selectionmode. The camera 14 shoots real objects and the processor 24 recognizesthe real objects based on a camera image generated by shooting. Forexample, the real objects are recognized by an image recognitiontechnology using Vision API framework. The processor 24 causes thedisplay 12 to display a recognition result. FIG. 3 illustrates a displayexample.

A display screen 32 illustrated in FIG. 3 is viewed by the user throughthe display 12 of the AR glasses 10. A notebook PC 26, a sticky note 28,and a mouse 30 are real objects. The processor 24 recognizes thenotebook PC 26 and the mouse 30 as candidates for the first referenceobject. The processor 24 displays symbols 34 and 36 on the displayscreen 32 while laying the symbols 34 and 36 over the real space. Thesymbols 34 and 36 indicate the candidates for the first referenceobject. The symbol 34 is an image indicating the notebook PC 26. Thesymbol 36 is an image indicating the mouse 30.

The user selects the first reference object from among the recognizedreal objects by gesture, voice, or operation for the UI 20. Theprocessor 24 sets the selected real object as the first referenceobject. For example, in response to the user selecting the notebook PC26, the processor 24 sets the notebook PC 26 as the first referenceobject and stores an image of the notebook PC 26 in the memory 22 as theimage of the first reference object. The processor 24 may set this firstreference object as a default first reference object.

The images of the real objects (e.g., the notebook PC 26, the stickynote 28, and the mouse 30) shot by the camera 14 may be displayed on thedisplay 12 and the user may select the image of the first referenceobject from among the displayed images. The processor 24 sets the realobject in the selected image as the first reference object.

As another example, the image of the first reference object may beprestored in a server and the processor 24 may recognize the firstreference object based on the image. For example, the image of the firstreference object is prestored in the server. The image of the firstreference object is linked to identification information of the firstreference object. The processor 24 recognizes the first reference objectbased on the identification information.

The processor 24 acquires an image of the target object from the camera14 (S02). The processor 24 may recognize, as the target object, anobject in contact with the first reference object, an object near thefirst reference object (e.g., an object within a predetermined distancerange from the position of the first reference object), an objectselected by the user, or an object other than the first reference objectwithin a view field of the camera 14. The processor 24 stores the imageof the target object in the memory 22.

Next, the processor 24 determines a relative positional relationshipbetween the first reference object and the target object based on thecamera image generated by the camera 14 (S03). For example, the cameraimage shows the first reference object and the target object and theprocessor 24 analyzes the camera image to determine the relativepositional relationship between the first reference object and thetarget object in the camera image. For example, the processor 24determines a position of the target object relative to the position ofthe first reference object in the camera image. The processor 24 storesinformation indicating the relative positional relationship (e.g.,information indicating the position of the target object relative to theposition of the first reference object) in the memory 22.

Next, the processor 24 generates a virtual object representing thetarget object based on the image of the target object (S04). If thetarget object contains a text or symbol, the processor 24 may recognizethe text or symbol by recognizing characters in the image of the targetobject through optical character recognition (OCR) or the like. Thevirtual object representing the target object may be the image of thetarget object, or a schematic image representing the target object. Theprocessor 24 stores the virtual object in the memory 22. If the text orsymbol is recognized, the processor 24 stores the virtual object in thememory 22 together with the recognition result.

For example, in response to selection of the sticky note 28 in FIG. 3 asthe target object, the processor 24 generates a virtual objectrepresenting the sticky note 28. The processor 24 may recognize a textor symbol on the sticky note 28 through character recognition andgenerate a virtual object containing the recognition result.

A virtual object display process is described below with reference toFIG. 4. FIG. 4 is a flowchart illustrating the process.

First, the camera 14 of the AR glasses 10 shoots an image and theprocessor 24 analyzes the camera image generated by the camera 14 todetermine whether a second reference object identical to the firstreference object is in a view field of the AR glasses 10. For example,the processor 24 processes the camera image by pattern matching (e.g.,OpenCV) to determine whether the camera image shows the second referenceobject identical to the first reference object (i.e., whether the secondreference object identical to the first reference object is in the viewfield of the AR glasses 10). The image of the first reference object isstored in the memory 22 and the processor 24 processes the camera imageby pattern matching using the image of the first reference object todetermine whether the camera image shows the second reference objectidentical to the first reference object.

If the second reference object identical to the first reference objectis not in the view field of the AR glasses 10 (S10: No), the process isterminated.

If the second reference object identical to the first reference objectis in the view field of the AR glasses 10 (S10: Yes), the processor 24determines whether the target object is in the view field of the ARglasses 10. The image of the target object is stored in the memory 22and the processor 24 processes the camera image by pattern matchingusing the image of the target object to determine whether the cameraimage shows the target object (i.e., whether the target object is in theview field of the AR glasses 10).

If the target object is not in the view field of the AR glasses 10 (S11:No), the processor 24 reflects the relative positional relationshipbetween the first reference object and the target object in a positionalrelationship between the second reference object and the target objectto be virtually displayed, and displays the virtual object representingthe target object on the screen of the display 12 while laying thevirtual object over the real space (S12). If the target object is not inthe view field of the AR glasses 10, the target object is not recognizedtogether with the second reference object identical to the firstreference object. In this case, the processor 24 displays the virtualobject representing the target object in relation to the secondreference object. More specifically, the processor 24 determines adisplay position of the virtual object representing the target object onthe screen of the display 12 relative to the position of the secondreference object (i.e., the object identical to the first referenceobject) that is in the view field of the AR glasses 10 in the real spacebased on the relative positional relationship between the firstreference object and the target object determined in Step S03. Then, theprocessor 24 displays the virtual object representing the target objectat the relative position on the screen of the display 12. The processor24 may hide or remove the virtual object from the screen in response toa user's instruction (e.g., gesture or voice).

If the target object is in the view field of the AR glasses 10 (S11:Yes), the processor 24 does not display the virtual object butdetermines whether details of the target object differ from details ofthe virtual object representing the target object. If the target objectis in the view field of the AR glasses 10 and recognized together withthe second reference object, the processor 24 does not display thevirtual object. The processor 24 determines whether the details of thetarget object shot and recognized together with the second referenceobject differ from the details of the virtual object representing thetarget object shot and recognized together with the first referenceobject in Step S02.

If there is no difference (S13: No), the process is terminated.

If there is the difference (S13: Yes), the processor 24 notifies theuser that the details have the difference (S14). For example, theprocessor 24 may cause the display 12 to display information indicatingthat the details have the difference, or cause the display of the UI 20to display the information. The processor 24 may notify the user thatthe details have a difference if the degree of the difference is equalto or higher than a predetermined threshold. For example, the processor24 determines that the details have a difference if the sticky noteserving as the target object is edited between the time when the stickynote is shot together with the first reference object and the time whenthe sticky note is shot together with the second reference object. Then,the processor 24 notifies the user that the details have the difference.The processor 24 may notify the user that the details have a differenceif the shapes differ from each other.

The notified user may choose whether to display the virtual objectrepresenting the target object. In response to the user giving aninstruction to display the virtual object by gesture, voice, oroperation for the UI 20, the processor 24 displays the virtual objectrepresenting the target object similarly to the process of Step S12.

The processor 24 may notify the user that the details have a differencebefore the virtual object representing the target object is displayed,after the virtual object representing the target object is displayed, orsimultaneously with the display of the virtual object representing thetarget object.

A process to be performed when the details have a difference isdescribed below in detail.

If a single virtual object represents the target object, that is, if thenumber of target objects is one and a virtual object representing thetarget object is generated, the processor 24 causes the display 12 todisplay, for example, the name of the target object and a message forinquiring of the user whether to display the virtual object representingthe target object. In response to a user's display instruction (e.g.,selection of “OK”), the processor 24 causes the display 12 to displaythe virtual object. In response to a user's cancellation instruction,the processor 24 prevents the display 12 from displaying the virtualobject. For example, the “OK” or cancellation instruction may be givenby gesture, voice, or operation for the UI 20.

If a plurality of target objects are present, the processor 24 causesthe display 12 to display a list of the plurality of target objects. Forexample, as illustrated in FIG. 5, a list 38 is displayed on the displayscreen 32. In response to the user selecting a target object from thelist 38 and giving a display instruction (e.g., selection of “OK”), theprocessor 24 causes the display 12 to display a virtual objectrepresenting the selected target object. For example, the selection ofthe virtual object and the “OK” instruction may be made by gesture,voice, or operation for the UI 20.

If a plurality of target objects are shot and recognized together withthe first reference object and if the plurality of target objects arenot in the view field of the AR glasses 10 when the second referenceobject is shot and recognized (S11: No), the processor 24 may display alist for selection of a target object to be virtually displayed (e.g.,the list 38) on the display screen 32. The processor 24 causes thedisplay 12 to display a virtual object representing the selected targetobject.

EXAMPLE 2

Example 2 is described below.

A process for acquiring images of the first reference object and thetarget object is described with reference to FIG. 6. FIG. 6 is aflowchart illustrating the process.

Similarly to Step S01, the camera 14 of the AR glasses 10 shoots animage and the processor 24 acquires an image of the first referenceobject from the camera 14 (S20). Similarly to Example 1, the user mayselect the first reference object, or the processor 24 may recognize anobject viewed by the user for the longest time as the first referenceobject. The image of the first reference object is stored in the memory22.

Next, the processor 24 identifies a category of the first referenceobject based on the image of the first reference object (S21). Forexample, the category of the first reference object is identified byrecognizing the first reference object by an image recognitiontechnology using Vision API framework. Specifically, a notebook PC, adesktop PC, a tablet PC, or a desk is identified as the category of thefirst reference object. Information indicating the category of the firstreference object is linked to the image of the first reference objectand stored in the memory 22.

Next, the processor 24 links the first reference object to a user ID ofthe user of the AR glasses 10 (S22). For example, the user of the ARglasses 10 logs into the AR glasses 10, the information processingapparatus 16, or a cloud service. For example, the user ID is linked tothe image of the first reference object and stored in the memory 22. Theuser ID may be managed by a server or the like.

Similarly to Step S02, the processor 24 acquires an image of the targetobject from the camera 14 (S23). The processor 24 stores the image ofthe target object in the memory 22.

Similarly to Step S03, the processor 24 determines a relative positionalrelationship between the first reference object and the target objectbased on the camera image generated by the camera 14 (S24). Informationindicating the relative positional relationship is stored in the memory22.

Similarly to Step SO4, the processor 24 generates a virtual objectrepresenting the target object based on the image of the target object(S25). The virtual object is stored in the memory 22.

A virtual object display process is described below with reference toFIG. 7. FIG. 7 is a flowchart illustrating the process.

The following process is executed when the user logs in and wears the ARglasses 10. For example, the user logs into the AR glasses 10, theinformation processing apparatus 16, or a cloud service.

The camera 14 of the AR glasses 10 shoots an image and the processor 24analyzes the camera image generated by the camera 14 to determinewhether a second reference object in a category identical to that of thefirst reference object is in the view field of the AR glasses 10. Thesecond reference object in a category identical to that of the firstreference object is an example of the object similar to the firstreference object. For example, the processor 24 processes the cameraimage by pattern matching (e.g., OpenCV) to determine whether the cameraimage shows the second reference object in a category identical to thatof the first reference object (i.e., whether the second reference objectin a category identical to that of the first reference object is in theview field of the AR glasses 10). The image of the first referenceobject and the information indicating the category of the firstreference object are stored in the memory 22 and the processor 24processes the camera image by pattern matching using the image of thefirst reference object to determine whether the camera image shows thesecond reference object in a category identical to that of the firstreference object.

If the second reference object in a category identical to that of thefirst reference object is not in the view field of the AR glasses 10(S30: No), the process is terminated.

If the second reference object in a category identical to that of thefirst reference object is in the view field of the AR glasses 10 (S30:Yes), the processor 24 determines, similarly to Step S11, whether thetarget object is in the view field of the AR glasses 10. The image ofthe target object is stored in the memory 22 and the processor 24processes the camera image by pattern matching using the image of thetarget object to determine whether the camera image shows the targetobject (i.e., whether the target object is in the view field of the ARglasses 10).

If the target object is not in the view field of the AR glasses 10 (S31:No), the processor 24 executes a positional relationship correctionprocess while the second reference object in a category identical tothat of the first reference object is regarded as the first referenceobject (S32), reflects the relative positional relationship between thefirst reference object and the target object in a relative positionalrelationship between the second reference object and the target objectto be virtually displayed, and displays the virtual object representingthe target object on the screen of the display 12 while laying thevirtual object over the real space (S33). If the target object is not inthe view field of the AR glasses 10, the target object is not recognizedtogether with the second reference object in a category identical tothat of the first reference object. In this case, the processor 24displays the virtual object representing the target object in relationto the second reference object. The correction process involves changingthe size of the virtual object representing the target object dependingon the ratio between the size of the first reference object and the sizeof the second reference object, and changing a display position of thevirtual object representing the target object relative to the secondreference object. A specific example of this virtual process isdescribed later in detail.

If the target object is in the view field of the AR glasses 10 (S31:Yes), the processor 24 determines whether details of the target objectdiffer from details of the virtual object representing the targetobject.

If there is no difference (S34: No), the process is terminated.

If there is the difference (S34: Yes), the processor 24 notifies theuser that the details have the difference similarly to Step S14 (S35).Similarly to Example 1, the notified user may give an instruction todisplay the virtual object representing the target object. In responseto the user giving the instruction to display the virtual object, theprocessor 24 displays the virtual object representing the target object.

For example, if the first reference object is a PC (A) and the secondreference object is a notebook PC (B), the second reference object isnot identical to the first reference object but is in a categoryidentical to that of the first reference object. If the notebook PC (B)serving as the second reference object is in the view field of the ARglasses 10 and if the target object is not in the view field of the ARglasses 10, the virtual object representing the target object is laidover the real space.

EXAMPLE 3

Example 3 is described below.

A process for acquiring images of the first reference object and thetarget object is described with reference to FIG. 8. FIG. 8 is aflowchart illustrating the process.

Similarly to Step S01, the camera 14 of the AR glasses 10 shoots animage and the processor 24 acquires an image of the first referenceobject from the camera 14 (S40). In Example 3, the processor 24 acquiresimages of a plurality of first reference objects. The images of thefirst reference objects are stored in the memory 22.

A process for acquiring the images of the plurality of first referenceobjects is described below in detail. The user may select the pluralityof first reference objects, or the processor 24 may recognize the firstreference objects in descending order of the length of time during whichthe user views the objects.

An example of selection of the plurality of reference objects by theuser is described below.

In response to the user giving an instruction to execute the referenceobject selection mode by gesture, voice, or operation for the UI 20, theinformation processing system enters the reference object selectionmode. The camera 14 shoots real objects and the processor 24 recognizesthe real objects based on a camera image generated by shooting. Forexample, the real objects are recognized by an image recognitiontechnology. The processor 24 causes the display 12 to display arecognition result. FIG. 9 illustrates a display example.

A display screen 32 illustrated in FIG. 9 is viewed by the user throughthe display 12 of the AR glasses 10. A notebook PC 26, a sticky note 28,and a mouse 30 are real objects. The processor 24 recognizes thenotebook PC 26, the mouse 30, and a keyboard of the notebook PC 26 ascandidates for the first reference object. The processor 24 displayssymbols 34, 36, and 40 on the display screen 32 while laying the symbols34, 36, and 40 over the real space. The symbols 34, 36, and 40 indicatethe candidates for the first reference object. The symbol 34 is an imageindicating the notebook PC 26. The symbol 36 is an image indicating themouse 30. The symbol 40 is an image indicating the keyboard of thenotebook PC 26.

The user selects the first reference object from among the recognizedreal objects by gesture, voice, or operation for the UI 20. Theprocessor 24 sets the selected real object as the first referenceobject. For example, in response to the user selecting the notebook PC26, the processor 24 sets the notebook PC 26 as the first referenceobject and stores an image of the notebook PC 26 in the memory 22 as theimage of the first reference object. The user may select another realobject as the first reference object. For example, in response to theuser selecting the keyboard of the notebook PC 26 and the mouse 30, theprocessor 24 sets the keyboard of the notebook PC 26 and the mouse 30 asthe first reference objects and stores an image of the keyboard of thenotebook PC 26 and an image of the mouse 30 in the memory 22 as theimages of the first reference objects. The processor 24 may set theorder of selection as ordinal levels of the first reference objects.

In response to the user giving an instruction to terminate the referenceobject selection mode by gesture, voice, or operation for the UI 20, thereference object selection mode is terminated. Then, the plurality ofselected real objects are set as the first reference objects.

Next, the processor 24 determines ordinal levels of the plurality offirst reference objects (S41). The user may set the ordinal levels ofthe first reference objects, or the processor 24 may determine theordinal levels of the first reference objects in descending order of thelength of time during which the user views the first reference objects.

Similarly to Step S02, the processor 24 acquires an image of the targetobject from the camera 14 (S42). The processor 24 stores the image ofthe target object in the memory 22.

Similarly to Step S03, the processor 24 determines a relative positionalrelationship between the first reference object and the target objectbased on the camera image generated by the camera 14 (S43). Informationindicating the relative positional relationship is stored in the memory22. In Example 3, the plurality of first reference objects are set andtherefore the processor 24 determines relative positional relationshipsbetween the individual first reference objects and the target object.

Similarly to Step S04, the processor 24 generates a virtual objectrepresenting the target object based on the image of the target object(S44). The virtual object is stored in the memory 22.

A virtual object display process is described below with reference toFIG. 10. FIG. 10 is a flowchart illustrating the process.

The camera 14 of the AR glasses 10 shoots an image and the processor 24analyzes the camera image generated by the camera 14 to determinewhether a second reference object identical to at least one firstreference object out of the plurality of first reference objects is inthe view field of the AR glasses 10.

If no second reference object identical to the first reference object isin the view field of the AR glasses 10 (S50: No), the process isterminated.

If the second reference object identical to at least one first referenceobject is in the view field of the AR glasses 10 (S50: Yes), theprocessor 24 determines a second reference object for use in thesubsequent process based on the ordinal levels determined in Step S41(S51). For example, the processor 24 determines an object identical to afirst reference object having a higher ordinal level as the secondreference object for use in the subsequent process.

Similarly to Step S11, the processor 24 determines whether the targetobject is in the view field of the AR glasses 10.

If the target object is not in the view field of the AR glasses 10 (S52:No), the processor 24 displays, similarly to Step S12, the virtualobject representing the target object in relation to the secondreference object determined in Step S51.

If the target object is in the view field of the AR glasses 10 (S52:Yes), the processor 24 determines, similarly to Step S13, whetherdetails of the target object differ from details of the virtual objectrepresenting the target object.

If there is no difference (S54: No), the process is terminated.

If there is the difference (S54: Yes), the processor 24 notifies theuser that the details have the difference similarly to Step S14 (S55).Similarly to Example 1, the notified user may give an instruction todisplay the virtual object representing the target object. In responseto the user giving the instruction to display the virtual object, theprocessor 24 displays the virtual object representing the target object.

The processor 24 may determine whether a second reference object similarto at least one first reference object out of the plurality of firstreference objects is in the view field of the AR glasses 10. If thesecond reference object similar to the first reference object is in theview field of the AR glasses 10, the positional relationship correctionprocess is executed and the virtual object representing the targetobject is displayed in relation to the second reference object.

A specific example of the first exemplary embodiment is described belowwith reference to FIG. 11 and FIG. 12. FIG. 11 schematically illustratesreal objects in the real space. For example, the AR glasses 10, thenotebook PC 26, and the sticky note 28 are illustrated as the realobjects. FIG. 12 illustrates the display screen 32 of the AR glasses 10.

As illustrated in FIG. 11, for example, the camera 14 of the AR glasses10 shoots the notebook PC 26 and the sticky note 28. The sticky note 28is in contact with the notebook PC 26. The notebook PC 26 is recognizedas the first reference object and an image of the notebook PC 26 isstored in the memory 22. The sticky note 28 is recognized as the targetobject and an image of the sticky note 28 is stored in the memory 22. Avirtual object representing the sticky note 28 is generated and storedin the memory 22. For example, the user selects the notebook PC 26 asthe first reference object and the sticky note 28 as the target object.

As illustrated in FIG. 12, for example, the user moves and a notebook PC42 is in the view field of the AR glasses 10. The camera 14 shoots thenotebook PC 42. The notebook PC 42 is a real object. The notebook PC 42is an object different from the notebook PC 26 serving as the firstreference object but is in a category identical to that of the notebookPC 26, that is, similar to the notebook PC 26. The sticky note 28 is notin contact with the notebook PC 42. Since the notebook PC 42 is anobject in a category identical to that of the notebook PC 26, theprocessor 24 recognizes the notebook PC 42 as the second referenceobject. Since the sticky note 28 is not in contact with the notebook PC42, that is, the sticky note 28 serving as the target object is not inthe view field of the AR glasses 10, the processor 24 displays a virtualobject 44 representing the sticky note 28 on the display screen 32. Atthis time, the processor 24 executes the correction process of Step S32to reflect a relative positional relationship between the notebook PC 26and the sticky note 28 in a relative positional relationship between thenotebook PC 42 and the virtual object 44.

The correction process is described below in detail with reference toFIG. 13 and FIG. 14. FIG. 13 schematically illustrates real objects inthe real space. For example, the notebook PC 26 serving as the firstreference object and the sticky note 28 serving as the target object areillustrated as the real objects. FIG. 14 schematically illustrates areal object and a virtual object. For example, the notebook PC 42serving as the second reference object and the virtual object 44representing the sticky note 28 are illustrated. The camera 14 of the ARglasses 10 shoots the notebook PC 42 serving as the second referenceobject and the virtual object 44 representing the sticky note 28 isdisplayed in relation to the notebook PC 42. The sticky note 28 is incontact with a display of the notebook PC 26.

First, the camera 14 shoots the notebook PC 26 and the sticky note 28and the processor 24 extracts a shape (e.g., a rectangle) of a portionin contact with the sticky note 28 (e.g., the display of the notebook PC26) and a shape (e.g., a rectangle) of the sticky note 28 from a cameraimage generated by shooting.

Next, the processor 24 sets reference points on the display of thenotebook PC 26 and the sticky note 28 in the camera image showing thenotebook PC 26 and the sticky note 28. For example, if the sticky note28 is provided on a right part of the display, the processor 24 sets areference point A1 at the upper right corner of the display and areference point B1 (x1, y1) at the upper left corner of the sticky note28. The coordinate x1 of the reference point B1 is a coordinate on an Xaxis in a case where the reference point A1 is the origin. Thecoordinate y1 of the reference point B1 is a coordinate on a Y axis inthe case where the reference point A1 is the origin. The X axis and theY axis are orthogonal to each other and coordinates are defined in thecamera image.

The processor 24 calculates a length L1 of the display of the notebookPC 26 on the Y axis and a length L2 of the display on the X axis basedon the camera image showing the notebook PC 26 and the sticky note 28.

Next, the camera 14 shoots the notebook PC 42 and the processor 24 setsa reference point on the notebook PC 42 in a camera image generated byshooting. For example, the processor 24 sets a reference point A2 at theupper right corner of a display of the notebook PC 42.

The processor 24 calculates a length L3 of the display of the notebookPC 42 on the Y axis and a length L4 of the display on the X axis basedon the camera image showing the notebook PC 42.

Next, the processor 24 calculates, based on the following expressions, areference point B2 (x2, y2) on the virtual object in a case where thereference point A2 is the origin. The reference point B2 corresponds tothe reference point B1 on the sticky note 28.

x2=L4/L2×x1

y2=L3/L1×y1

The coordinate x2 of the reference point B2 is a coordinate on the Xaxis in the case where the reference point A2 is the origin. Thecoordinate y2 of the reference point B2 is a coordinate on the Y axis inthe case where the reference point A2 is the origin.

Similarly to the positional relationship, the processor 24 may reflect arelative size relationship between the first reference object and thetarget object in a relative size relationship between the secondreference object and the virtually displayed target object (i.e., thevirtual object). In the specific example described above, the processor24 sets the reference point B2 at the upper left end and displays, onthe display screen 32, the virtual object 44 that is α times as large asthe sticky note 28.

The value α is a smaller one of (L4/L2) or (L3/L1).

By defining the value α in this manner, the virtual object is displayedin relation to the second reference object while keeping the relativesize relationship between the first reference object and the targetobject.

If the size of the target object is smaller than a threshold, theprocessor 24 need not reflect the relative size relationship between thefirst reference object and the target object in the relative sizerelationship between the second reference object and the virtual objectrepresenting the target object. In this case, the processor 24 maydisplay a virtual object having a predetermined minimum size, or avirtual object having an absolute value of the size of the shot targetobject.

Second Exemplary Embodiment

An information processing system according to a second exemplaryembodiment is described below with reference to FIG. 15. FIG. 15illustrates an example of the hardware configuration of the informationprocessing system according to the second exemplary embodiment.

The information processing system according to the second exemplaryembodiment includes a camera 46 and a server 48 in addition to the ARglasses 10 and the information processing apparatus 16. The informationprocessing apparatus 16 may be installed in the AR glasses 10 or theserver 48. Examples of the camera 46 include a stationary observationcamera installed at a predetermined location.

A process for acquiring images of the first reference object and thetarget object is described below with reference to FIG. 16. FIG. 16 is aflowchart illustrating the process.

First, the server 48 manages the camera 46 and the user ID by linkingthe camera 46 and the user ID together (S60). The link is made inadvance. The link may be managed by the information processing apparatus16.

Next, the camera 46 shoots the first reference object and an image ofthe first reference object is acquired (S61). For example, the image ofthe first reference object is stored in the server 48. The image of thefirst reference object may be stored in the memory 22 of the informationprocessing apparatus 16.

Next, the processor 24 identifies a category of the first referenceobject based on the image of the first reference object (S62). If theimage of the first reference object is stored in the server 48, theprocessor 24 acquires the image of the first reference object from theserver 48 and identifies the category of the first reference object. Theprocessor 24 links information indicating the category to the user ID.The information indicating the category may be stored in the server 48or the memory 22 of the information processing apparatus 16.

Next, the camera 46 shoots the target object and an image of the targetobject is acquired (S63). The image of the target object may be storedin the server 48 or the memory 22 of the information processingapparatus 16. A virtual object representing the target object isgenerated based on the image of the target object. The virtual objectmay be generated by the processor 24 or the server 48. The virtualobject is stored in the server 48.

Next, the processor 24 acquires a camera image generated by the camera46 and determines a relative positional relationship between the firstreference object and the target object based on the camera image (S64).

For example, the pieces of information (e.g., the camera image, theimage of the first reference object, the image of the target object, thevirtual object, and the information indicating the relative positionalrelationship) are periodically transmitted to and stored in the server48 (S65).

A virtual object display process is described below with reference toFIG. 17. FIG. 17 is a flowchart illustrating the process.

The following process is executed when the user logs in and wears the ARglasses 10. For example, the user logs into the AR glasses 10, theinformation processing apparatus 16, or the server 48.

The camera 14 of the AR glasses 10 shoots an image and the processor 24analyzes the camera image generated by the camera 14 to determinewhether a second reference object in a category identical to that of thefirst reference object is in the view field of the AR glasses 10.

If the second reference object in a category identical to that of thefirst reference object is not in the view field of the AR glasses 10(S70: No), the process is terminated.

If the second reference object in a category identical to that of thefirst reference object is in the view field of the AR glasses 10 (S70:Yes), the processor 24 acquires the virtual object from the server 48(S71).

If the target object is not in the view field of the AR glasses 10 (S72:No), the processor 24 executes the positional relationship correctionprocess similarly to Step S32 while the second reference object in acategory identical to that of the first reference object is regarded asthe first reference object (S73). Then, the processor 24 reflects therelative positional relationship between the first reference object andthe target object in a relative positional relationship between thesecond reference object and the virtual object and displays the virtualobject on the screen of the display 12 while laying the virtual objectover the real space (S74).

If the target object is in the view field of the AR glasses 10 (S72:Yes), the processor 24 determines whether details of the target objectdiffer from details of the virtual object representing the targetobject.

If there is no difference (S75: No), the process is terminated.

If there is the difference (S75: Yes), the processor 24 notifies theuser that the details have the difference similarly to Step S14 (S76).Similarly to Example 1, the notified user may give an instruction todisplay the virtual object representing the target object. In responseto the user giving the instruction to display the virtual object, theprocessor 24 displays the virtual object representing the target object.

A specific example of the second exemplary embodiment is described belowwith reference to FIG. 18 and FIG. 19. FIG. 18 schematically illustratesreal objects in the real space. For example, the camera 46, the notebookPC 26, the sticky note 28, a desk 50, and paper 52 are illustrated asthe real objects. FIG. 19 illustrates the display screen 32 of the ARglasses 10.

As illustrated in FIG. 18, for example, the camera 46 shoots thenotebook PC 26, the sticky note 28, the desk 50, and the paper 52. Forexample, the camera 46 shoots an image in an office. The sticky note 28is in contact with the notebook PC 26. The paper 52 is not in contactwith the notebook PC 26 but is near the notebook PC 26. For example, thenotebook PC 26 is recognized as the first reference object and an imageof the notebook PC 26 is stored in the server 48. The sticky note 28 andthe paper 52 are recognized as the target objects and images of thesticky note 28 and the paper 52 are stored in the server 48. Virtualobjects representing the sticky note 28 and the paper 52 are generatedand stored in the server 48. For example, the user selects the notebookPC 26 as the first reference object and the sticky note 28 and the paper52 as the target objects.

As illustrated in FIG. 19, for example, the user wears the AR glasses 10in his/her home or in a cafe and the notebook PC 42 is in the view fieldof the AR glasses 10. The camera 14 of the AR glasses 10 shoots thenotebook PC 42. The notebook PC 42 is a real object. The notebook PC 42is an object different from the notebook PC 26 serving as the firstreference object but is in a category identical to that of the notebookPC 26. The sticky note 28 is not in contact with the notebook PC 42 andthe paper 52 is not near the notebook PC 42. Since the notebook PC 42 isan object in a category identical to that of the notebook PC 26, theprocessor 24 recognizes the notebook PC 42 as the second referenceobject. Since the sticky note 28 is not in contact with the notebook PC42, that is, the sticky note 28 serving as the target object is not inthe view field of the AR glasses 10, the processor 24 displays thevirtual object 44 representing the sticky note 28 on the display screen32. At this time, the processor 24 executes the correction process toreflect a relative positional relationship between the notebook PC 26and the sticky note 28 in a relative positional relationship between thenotebook PC 42 and the virtual object 44. Since the paper 52 serving asthe target object is not in the view field of the AR glasses 10, theprocessor 24 displays a virtual object 54 representing the paper 52 onthe display screen 32. At this time, the processor 24 executes thecorrection process to reflect a relative positional relationship betweenthe notebook PC 26 and the paper 52 in a relative positionalrelationship between the notebook PC 42 and the virtual object 54.

In the embodiments above, the term “processor” refers to hardware in abroad sense. Examples of the processor include general processors (e.g.,CPU: Central Processing Unit), and dedicated processors (e.g., GPU:Graphics Processing Unit, ASIC: Application Specific Integrated Circuit,FPGA: Field Programmable Gate Array, and programmable logic device). Inthe embodiments above, the term “processor” is broad enough to encompassone processor or plural processors in collaboration which are locatedphysically apart from each other but may work cooperatively. The orderof operations of the processor is not limited to one described in theembodiments above, and may be changed.

The foregoing description of the exemplary embodiments of the presentdisclosure has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit thedisclosure to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiments were chosen and described in order to best explain theprinciples of the disclosure and its practical applications, therebyenabling others skilled in the art to understand the disclosure forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of thedisclosure be defined by the following claims and their equivalents.

What is claimed is:
 1. An information processing apparatus comprising aprocessor configured to recognize a first real reference object in areal space, and if a second real reference object identical or similarto the first real reference object is recognized in the real space,display a virtual version of a real target object on the second realreference object, wherein the real target object which is touching thefirst real reference object is recognized in the real space in advancetogether with the first real reference object.